Learning Objectives

After completing this lesson, you’ll be able to:

Web-Based Datasets

There can be no doubt about the trend toward storing data in the cloud, including spatial data. For that reason, FME has comprehensive tools for reading web-based datasets.

Note

Pure web data is usually accessed via APIs and returned as JSON or XML. FME supports reading that kind of data, but in this course, we focus on reading file formats stored on the web.

Simple URL Selection

The easiest way to read a web-based dataset is to paste the URL into the source Dataset parameter.

Reading from a URL

Here the workspace author is adding a reader to read a MapInfo TAB dataset directly from an open data catalog stored on Amazon S3.

Note

A folder-based dataset must be compressed to a single zip file for FME to read it from the web like this; the above perfectly illustrates that requirement. Datasets can be read from non-zipped datasets, but only when the dataset consists of a single file (such as an AutoCAD DWG file).

The URL entered into a source dataset field may refer to any resource on a web-based file storage system. For example, here, a user is reading the same MapInfo TAB dataset directly from a Dropbox link:

Reading from a Dropbox URL

The key is forcing the web service to provide a direct link to the file rather than its web interface. For example, to force Dropbox to provide a direct link, you should set the dl query parameter to 1 in the URL, as in the above screenshot, instead of the default value of 0 (zero).

If data is read from a website data needing authentication, most readers have parameters to enter such information:

Use Network Authentication parameter

However, there are better ways to read data from a web service.

Web Services

Besides being able to read from a URL, FME can directly access certain web services to read data. Direct access to a web service is achieved with functionality inside FME called web connections.

Web Connections are created by selecting Tools > FME Options > Web Connections on the FME Workbench menu bar and clicking the plus button in the Web Connections dialog:

Adding a Web Connection

New web service support is added regularly to FME. With the new FME packages system, web connections can be updated without needing to update all of FME. You can also define custom web connections and share them with other users on the FME Hub.

FME supports connections for a large number of web services - and the Manage Services button allows you to integrate any other web service you require - but some key ones are:

These are key types because they are capable of storing data in a way that can be accessed directly from the Add Reader dialogs:

Select File from Web dialog

In the above screenshot, an author is adding a MapInfo dataset from Amazon S3. This action opens a dialog in which they can browse Amazon S3 for the required file:

Selecting data from S3

The reader is then added to the workspace and functions like any other.

Note that if you wish to use a FeatureReader transformer instead of a reader, then the same "Select File From Web" option is available.

Note

Usually, you need to authenticate when adding a new web service connection into FME. Suppose the workspace is published to FME Server or copied to another FME Desktop installation. In that case, you must re-authenticate the new location. You can also make database and web connections public.

Connector Transformers

Besides being able to direct a reader to a web service, various transformers exist that can read files. These are similar to the list supported by readers:

AzureBlobStorageConnector DropboxConnector ProjectWiseWSGConnector
AzureFileStorageConnector FMEServerResourceConnector S3Connector
AutodeskA360Connector GoogleDriveConnector SalesForceConnector
BoxConnector HDFSConnector SlackConnector
CesiumIonConnector OneDriveConnector  

These transformers don't read data in the traditional FME sense. Instead, they copy a selected file and either add its contents to an attribute or download the file to the local filesystem.

Here, for example, an author is using a Creator transformer to trigger the reading of a file from Google Drive:

Creator plus GoogleDriveConnector

FME adds the file's contents to an attribute you can process as required. For example, maybe it is a snippet of XML that can be decoded with an XML transformer.

Note

Connector transformers can also upload a file, list the contents of a web service, and delete files from that service.

Given the "Select File From Web" tool on readers, Connector transformers are not generally intended for reading source data. However, they could be used to download a file that is then read using a FeatureReader.

Another use would be to retrieve a list of files that are then read directly using a FeatureReader.

However, the more likely use is to transfer written data to the web. Writers don't have an equivalent "Select File From Web" option. The best alternative is to write data with a FeatureWriter transformer and then use a Connector transformer to transfer that data to a web service of choice.